4 research outputs found
Post-processing Private Synthetic Data for Improving Utility on Selected Measures
Existing private synthetic data generation algorithms are agnostic to
downstream tasks. However, end users may have specific requirements that the
synthetic data must satisfy. Failure to meet these requirements could
significantly reduce the utility of the data for downstream use. We introduce a
post-processing technique that improves the utility of the synthetic data with
respect to measures selected by the end user, while preserving strong privacy
guarantees and dataset quality. Our technique involves resampling from the
synthetic data to filter out samples that do not meet the selected utility
measures, using an efficient stochastic first-order algorithm to find optimal
resampling weights. Through comprehensive numerical experiments, we demonstrate
that our approach consistently improves the utility of synthetic data across
multiple benchmark datasets and state-of-the-art synthetic data generation
algorithms
On the Importance of Calibration in Semi-supervised Learning
State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been
highly successful in leveraging a mix of labeled and unlabeled data by
combining techniques of consistency regularization and pseudo-labeling. During
pseudo-labeling, the model's predictions on unlabeled data are used for
training and thus, model calibration is important in mitigating confirmation
bias. Yet, many SOTA methods are optimized for model performance, with little
focus directed to improve model calibration. In this work, we empirically
demonstrate that model calibration is strongly correlated with model
performance and propose to improve calibration via approximate Bayesian
techniques. We introduce a family of new SSL models that optimizes for
calibration and demonstrate their effectiveness across standard vision
benchmarks of CIFAR-10, CIFAR-100 and ImageNet, giving up to 15.9% improvement
in test accuracy. Furthermore, we also demonstrate their effectiveness in
additional realistic and challenging problems, such as class-imbalanced
datasets and in photonics science.Comment: 24 page
Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetries
Deep ensembles (DE) have been successful in improving model performance by
learning diverse members via the stochasticity of random initialization. While
recent works have attempted to promote further diversity in DE via
hyperparameters or regularizing loss functions, these methods primarily still
rely on a stochastic approach to explore the hypothesis space. In this work, we
present Multi-Symmetry Ensembles (MSE), a framework for constructing diverse
ensembles by capturing the multiplicity of hypotheses along symmetry axes,
which explore the hypothesis space beyond stochastic perturbations of model
weights and hyperparameters. We leverage recent advances in contrastive
representation learning to create models that separately capture opposing
hypotheses of invariant and equivariant functional classes and present a simple
ensembling approach to efficiently combine appropriate hypotheses for a given
task. We show that MSE effectively captures the multiplicity of conflicting
hypotheses that is often required in large, diverse datasets like ImageNet. As
a result of their inherent diversity, MSE improves classification performance,
uncertainty quantification, and generalization across a series of transfer
tasks.Comment: Camera Ready Revision. ICML 202
Constructive Assimilation: Boosting Contrastive Learning Performance through View Generation Strategies
Transformations based on domain expertise (expert transformations), such as
random-resized-crop and color-jitter, have proven critical to the success of
contrastive learning techniques such as SimCLR. Recently, several attempts have
been made to replace such domain-specific, human-designed transformations with
generated views that are learned. However for imagery data, so far none of
these view-generation methods has been able to outperform expert
transformations. In this work, we tackle a different question: instead of
replacing expert transformations with generated views, can we constructively
assimilate generated views with expert transformations? We answer this question
in the affirmative and propose a view generation method and a simple, effective
assimilation method that together improve the state-of-the-art by up to ~3.6%
on three different datasets. Importantly, we conduct a detailed empirical study
that systematically analyzes a range of view generation and assimilation
methods and provides a holistic picture of the efficacy of learned views in
contrastive representation learning.Comment: Accepted at Generative Models for Computer Vision Workshop 202